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WHAT IS CLAIMED IS: 

1. An information presentation apparatus 
comprising : 

user operation input unit, adapted to input an 
5 operation of a user; 

user viewpoint position and pose measurement 
unit, adapted to measure a position and pose at a 
user's viewpoint; 

model data storage unit, adapted to store 
10 virtual world model data, real world model data, and 
data necessary to generate a virtual world image; 

annotation data storage unit, adapted to store 
data necessary to be added to a real world and a 
virtual world and then displayed; 
15 virtual image generation unit, adapted to 

generate an image of the virtual world by using 
information in said user viewpoint position and pose 
measurement unit, said model data storage unit and 
said annotation data storage unit; 
20 user viewpoint image input unit, adapted to 

capture an image of the real world viewed from the 
user's viewpoint; and 

image display unit, adapted to display an image 
obtained by synthesizing the image generated by said 
25 virtual image generation unit and the image obtained 
by said user viewpoint image input unit, on an image 
display device of the user. 
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2. An information presentation apparatus 
according to Claim 1, wherein plural information 
presentation apparatuses are provided, and said 
information presentation apparatus is connected to 
5 other information presentation apparatus through a 

transmission channel so as to exchange communication 
data. 



3* An information presentation apparatus 
10 according to Claim 2, wherein the communication data 
includes an identification number of each user using 
said information presentation apparatus, a name for 
discriminating each user, position and pose 
information of each user's viewpoint, operation 
15 information of each user, and annotation data. 

4. An information presentation apparatus 
according to Claim 1, wherein the virtual world model 
data includes three-dimensional coordinates of 

20 vertices of a polygon of a virtual computer graphics 
(CG) object arranged on the virtual world, structure 
information of faces of the polygon, discrimination 
information of the CG object, color information, 
texture information, a size of the CG object, and 

25 position and pose information indicating the 

arrangement of the CG object on the virtual world. 
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5. An information presentation apparatus 
according to Claim 1, wherein the real world model 
data includes three-dimensional coordinates of 
vertices of a polygon of an object existing in the 

5 real world merged with the virtual world, structure 
information of faces of the polygon, discrimination 
information of the object, a size of the object, and 
position and pose information indicating the 
arrangement of the object. 

10 

6. An information presentation apparatus 
according to Claim 1, wherein the data necessary to 
generate the virtual world image includes internal 
parameters such as size and angle of an image pickup 

15 element of an image pickup device of said user 

viewpoint image input unit, an angle of view of a 
lens, a lens distortion parameter and the like. 

7. An information presentation apparatus 

20 according to Claim 1, wherein said annotation data 
storage unit can store annotation data being 
additional information to be displayed on the real 
world and the virtual world. 

25 8 . An information presentation apparatus 

according to Claim 7, wherein the annotation data 
includes position and pose information of an object 
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arranged on the real world and the virtual world, 
discrimination information of the object, and text, 
symbol and image information for indicating 
information of the object to the user. 

5 

9. An information presentation apparatus 
according to Claim 1, wherein said virtual image 
generation unit draws the information stored in said 
model data storage unit from the user's viewpoint in 

10 computer graphics to generate the image of the 

virtual world viewed from the user's viewpoint, by- 
using the position and pose information at the user's 
viewpoint obtained from said user viewpoint position 
and pose measurement unit. 

15 

10. Ah information presentation apparatus 
according to Claim 9, wherein said virtual image 
generation unit has a function to transfer data to a 
transmission channel and a function to receive data 

20 from the transmission channel. 

11. An information presentation apparatus 
according to Claim 9, wherein said the virtual image 
generation unit has a function to generate an 

25 annotation by selecting the information to be 

presented to the user from the annotation data stored 
in said annotation data storage unit on the basis of 
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the position and pose at the user's viewpoint 
obtained from said user viewpoint position and pose 
measurement unit and the position and pose of other 
user's viewpoint obtained through the transmission 
channel, and to superpose the generated annotation on 
the image of the virtual world. 

12 . An information presentation apparatus 
according to Claim 11, wherein the annotation 
includes a symbol, a character string, and image 
information . 

13. An information presentation apparatus 
according to Claim 9, wherein said virtual image 
generation unit has a function to automatically 
recognize a target that the user pays attention. 

14. An information presentation apparatus 
according to Claim 9, wherein said virtual image 
generation unit has a function to recognize a target 
that the user pays attention, by the user's operation 
input to said user operation input unit. 

15. An information presentation apparatus 
according to Claim 9, wherein said virtual image 
generation unit has a function, in a case where a 
target that other one or more users pay attention is 
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outside a visual range of the user, to generate an 
annotation indicating a direction of the target. 

16. An information presentation apparatus 
5 according to Claim 9, wherein said virtual image 

generation unit has a function, in a case where a 
target that other one or more users pay attention is 
inside a visual range of the user, to generate an 
annotation indicating information of the target. 

10 

17. An information presentation apparatus 
according to Claim 9, wherein said virtual image 
generation unit has a function to generate an 
annotation of which the attributes of a color, a 

15 shape and a character type have been changed in 

regard to each user, and an annotation indicating a 
name for discriminating the user. 

18. An information presentation apparatus 
20 according to Claim 9, wherein said virtual image 

generation unit has a function capable of controlling 
a generated annotation, by the user's operation input 
to said user operation input unit. 

25 19. An information presentation apparatus 

according to Claim 9, wherein said virtual image 
generation unit has a function to generate an 
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annotation indicating information of a target that 
the user pays attention, in a state that its 
attributes of a color, a shape and a character type 
have been different from those of other annotation. 

20. An information presentation apparatus 
according to Claim 9, wherein said virtual image 
generation unit has a function to generate an 
annotation indicating a direction of other user 
existing outside a visual range of the user. 

21. An information presentation apparatus 
according to Claim 9, wherein said virtual image 
generation unit has a function to generate an 
annotation indicating a position of other user 
existing inside a visual range of the user. 

22. An information processing method comprising 
the steps of: 

inputting viewpoint information of a user; 

generating a virtual world image according to 
the viewpoint information, by using previously held 
virtual world data; 

generating an annotation concerning an attention 
target; and 

generating ' an image obtained by synthesizing an 
image of a real world, generated virtual world image 
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and the generated annotation, 

23. An information processing method according 
to Claim 22, wherein, in a case where the attention 
5 target exists outside the synthesized image, an 

annotation indicating a direction of the attention 
target is generated and synthesized to the 
synthesized image . 

10 24. An information processing method according 

to Claim 22, wherein, in a case where the attention 
target exists inside the synthesized image, an 
annotation indicating additional information for the 
attention target and having an attribute different 

15 from that of other annotation is generated and 
synthesized to the synthesized image* 

25. An information processing method according 
to Claim 22, wherein an annotation indicating whether 

20 or not the attention target is being observed by 
other user is generated and synthesized to the 
synthesized image . 

26. An information processing method according 
25 to Claim 22, wherein an annotation indicating a 

position of other user is generated and synthesized 
to the synthesized image. 
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27 . A program to achieve an information 
processing method comprising the steps of: 

inputting viewpoint information of a user; 

generating a virtual world image according to 
5 the viewpoint information, by using previously held 
virtual world data; 

generating an annotation concerning an attention 
target; and 

generating an image obtained by synthesizing an 
10 image of a real world, generated virtual world image 
and the generated annotation. 



